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ABSTRACT 

Results of a number of experiments to Illuminate th© relative effective- 
ness and costs of computerised information retrieval In the Interactive mode 
are reported. It was found that for equal time spent In preparing the search 
strategy, the batch and interactive modes gave approximately equal recall and 
relevance. The Interactive mode however encourages the searcher to devot® 
more time to the task and therefor© usually yields Improved output. Engineer- 
ing costs as a result are higher In this mode. Estimates of associated hard- 
ware costa also Indicate that operation in this mod® Is more expensive. 

Skilled RECON users like the rapid feedback and additional features offered 
by this mode If they are not constrained by considerations of cost, 
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OBJECTIVES OF THE STUDY 


Upon acquisition of a RECON terminal, the North Carolina Science and 
Technology Research Center undertook a series of experiments designed to 
i i 1 umi nate 

). The effectiveness (as measured by recall and relevance) of 
the interactive search mode as compared with the batch search 
mode. 

2. The differences in costs between searches performed inter- 
actively and those performed in batch processes. 

3. The search methods employed and results achieved by searchers 
of varying background and searching experience. 


RESEARCH METHODS 


The last objective was attacked first. Each searcher was put before 
the cathode ray tube display and typewriter keyboard. A printer alongside 
was rigged to record the enti re ’’conversation," including clock time, be- 
tween the searcher and RECON. The searcher was handed a statement of the 
problem and asked to go to work. The room was semi-sealed against distur- 
bance. Only If the searcher had trouble making himself understood to the 
system (which sometimes happened with inexperienced searchers) or encoun- 
tered technical difficulty (which occurs at random times) did he emerge 
and ask assistance of monitoring personnel. 

The search problem selected was one received during the normal course 
of business at the North Carolina Science and Technology Research Center. 

It had proven to be particularly troublesome because the 400 or so rele- 
vant documents in the collection suffered from indexing vagaries that made 
their retrieval difficult without, at the same time, retrieving a large 
quantity of extraneous material. In such a problem, it was felt, a 
searcher could really demonstrate his skill and knowledge. The search 
problem statement is reproduced at the end of this section. The printed 
record of the "conversation" with the computer was given to the experiment 
director along with the "hit" list the searcher felt was his best effort. 
Abstracts for all of those "hits" were examined by the experiment director 
who decided which of the hits were relevant. This was done to determine 
what portion of the searcher’s hit list was pertinent and how many of the 
pertinent documents in the entire collection the search identified. 

Theprinted records of the "conversations" with the system were then 
prepared in the form of a standardized flow chart to facilitate identifying 
the approach and results of each searcher. In the process of preparation, 
mistakes which elicited responses from the system such as "Invalid Command ' 
Proceed" were eliminated as were garbled conversations resulting from sys- 
tem failures. Reproduced as Figure I (see end of this section) is the flow 



chart for a search performed by an undergraduate engineering student with 
a fair knowledge of the scientific fundamentals involved and a better knowl- 
edge of system operation and indexing problems. The search shown in Figure I 
required the searcher to remain at the terminal approximately three hours. 
Figures 2-8 present flow charts depicting the search procedure used in the 
other seven searches made on this same question. 

Several approaches were tried in order to develop data toward the 
first objective. The same individual was asked to make his search first by 
the batch process and then, before the results of his efforts were known, 
do it again on RECON. The outputs and strategies were then compared by the 
experiment director. A limited number of searches, done by one person on 
RECON and another by batch process, were tried but the results there seemed 
more dependent on individual approaches than on the differences between 
RECON and the NC/STRC batch search procedure. This approach was therefore 
discontinued. 

Costs differences between a batch process search and an interactive 
search were also very difficult to evaluate for a number of reasons: 

1. The line charges and computer charges per search on RECON 
are not directly known because of the way the system and 
its accounting are set up; 

2. An individual usually does not perform a search exactly 
the same way on the two systems. 

3. RECON experiences occasional shutdowns and frequent over- 
load ings of the system which result in long response times 
to questions and instructions. 

4. An engineer can and usually does obtain intermediate output 
during a RECON search which he uses to phrase his remaining 
questions. 

Although discussing the cost differences between the batch and inter- 
active search approaches was judged to be a difficult task for the reasons 
mentioned above, it was felt that STRC had enough specific accounting data 
on the batch system and could obtain enough data on the other RECON-type 
on-line search systems to make a worthwhile cost comparison between the two 
modes of searching. In addition, the experimental RECON searches provided 
sufficient data on the amount of human effort involved in the interactive 
search mode and STRC cost accounting reports provided the same information 
for batch searches. The resultant cost comparison can be found in another 
section of this report entitled "Relative Costs of Batch and Interactive 
Search Modes." 

In addition to these tests, members of the technical staff of NC/STRC 
used RECON as they wished during normal Center operations. As their skills 
and knowledge of the system improved, various members began to make in- 
creasing use of it. Included herein is a report of one subjective reac- 

tion to RECON - what he liked, what he did not like, and how he liked to 
use it. 
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SEARCH QUESTION 


"Airborne Thermal Mapping of Water Surfaces n 


A consulting firm is bidding on a contract to 
map the water temperature in lakes which are used 
to cool thermal power plants . The mapping will be 
done by flying over in an airplane (ZOO O' - 6000' 
above lake surface) , observing the lake with a 
radiometer , recording the radiometer output , and 
then processing the data so as to be able to draw 
lines of constant temperature on an accurate out - 
line of the lake. The purpose , of course, is to 
make it possible to check compliance with thermal 
pollution standards easily and quickly. 

Specifications call for measuring accuracies 
of 0.5°F. The consulting firm thinks that 1°F is 
about all that they can do realistically. So, 
they would like to know (a) does any responsible 
group claim Q.5°F. accuracy? (b) If so, what 

1. sensor technology 

2. mathematical modeling techniques 

Z. data reduction methods 

4. computerized data processing techniques 

do they employ? 

NOTE: A radiometer is a device which senses the 

heat radiated from the surface of a body. It can 
be focused so that it "looks" at a relatively 
small area. The atmosphere absorbs some of this 
radiation, the absorption being dependent upon 
air temperature, density , and distance the radia- 
tion travels. Therefore, the angle at which the 
radiometer looks, at the surface from the airplane 
determines how much air the heat waves have to go 
through, what its density is, and how it's " layered 

From this knowledge and a mathematical model 
of the absorption and dispersion of infrared radi- 
ation by a variable density atmosphere, one can 
arrive at an estimate of the surface temperature 
in terms of the radiometer indication. Also , a 
lake is only a semi-opaque body, which means that 
all of the radiation that one detects is from the 
layer immediately below the surface. These effects 
are also included in the mathematical model. 



The radiometer output ie an electrical signal 
proportional to the intensity of the thermal radi- 
atvon and vts spectral dbstrvbutvon Chow much energy 
is radiated at each wavelength) . One must there- 
fore have a procedure for converting this signal 
to temperature. Ideally t the whole process could 
be one in which the user merely submits a magnetic 
tape recording of radiometer output looking at a 
given part of the lake to a computer program. The 
program reduces the data from this and other passes 
over the lake t and from these maps the lake surface 
temperatures. 



RESULTS OF THE FIRST SEARCH PROBLEM 

The search question posed was first performed by FOS as a batch search. 
Several reruns and considerable expense were required to obtain a sufficient 
quantity of relevant output. Center management felt that it would there- 
fore be a good question on which to compare the effectiveness of various 
approaches to the question and the backgrounds of the searchers. The capa- 
bility of recording the dialog which the searcher had with the system made 
it possible to study in detail the logic used by each individual. From 
this dialog, more formalized flow charts were prepared. The seven flow 
charts are discussed in detail below. Following this is a correlation of 
the relevancy and recall of the various searches with search logic and 
searcher's background. Completing the data assembled from this experiment 
Is Table I (see Appendix). This lists the documents recovered from all 
searches which the experiment director called pertinent, the searches 
identifying that document, the number of pertinent documents identified 
by each search as a percentage of the total pertinent documents (recall) 
and as a percentage of the hits found by that search (relevance). Also 
shown in Table I are 10 additional pertinent documents found by the 
experiment director by constructing a search from the index terms appear- 
ing most often in the highly pertinent documents. This post experiment 
search had 103 hits of which 45 were pertinent. Thirty-five of these had 
been recovered during the experiment. 

In passing, it may be noted that manual review of each search output 
In sequence will usually lead to some inconsistencies in identifying docu- 
ments which had been identified as pertinent in one search, but were not 
so identified by the experiment director in his review of the search indi- 
cated. Between 6 % and 30 % of the pertinent hits in each search were so 
affected. In a topic of this nature many documents will be of at least 
peripheral interest to a broadly-trai ned, innovative researcher. However, 
as one moves further and further from the central thrust of the subject, 
discrimination becomes fuzzy. Whereas today a particular document with a 
tenuous relation to the subject might seem of possible Interest, tomorrow, 
it may not. As a result of this ambiguity, in subsequent comparisons of 
searcher effectiveness a master hit list was comp i led by keypunching the 
output of each search - along with an identifying code to indicate the 
source search - and then sorting. Only the abstracts from the master hit 
list were then examined to select the relevant documents. The relevant 
document accession numbers were then keypunched and- inserted in a computer 
program which gives the recall and. relevance for each individual search. 
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LOGICAL PROCESSES EMPLOYED BY SEARCHER WWS AS 
DEDUCED FROM HIS SEARCH 

i 

In order to follow the logical processes depicted in Figure I, one 
should note that the following symbolism is employed: 

(|) A rectangle indicates the term which was the basis for 
a following operation. 

(2) In the upper left corner of the rectangle is a symbol 

denoting the source of the term: 

m from his mind 

e from a previous expansion 

t from a thesaurus expansion 

c from a document's bibliographic citation 

(3) A rectangle with curved sides attached indicates what 
the searcher saw on the CRT screen, e.g., a "display." 

(4) The elongated hexagon denotes an operation, e.g., expand 
"thermal." 


(5) The diamond indicates a decision which can be answered 
yes or no. 

(6) The circle indicates premature termination of an operation. 

(7) Operations, display, etc., are arranged in chronological 
order with arrows leading from the proper antecedents and 
to the point in the search where subsequent operations are 
performed. 

(8) Where it is necessary to break a line at the bottom of a 
column an identifying letter is inserted so that when the 
reader again sees that letter he knows the line of which 
it is a continuation. 


(9) Lines to the right of terms in a display indicate that 
the searcher selected those terms to form sets. 

, , searcher began by requesting an expansion about the term he con- 

sfdered to be basic to the whole search, the term "thermal." On the basis 
of . firs+ display 1, he selected two terms and felt that these were 
sufficiently pertinent for him to continue the displays until the display 
limit was reached. From this expansion he selected a total of 9 terms 


Display refers to the material presented at a given time on the face 
of the cathode ray tube viewing device. 


Approximately 14 additional terms 
the command "more" is given unti I 


are displayed on the CRT each time 
the limit of 41 terms has been reached. 
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and formed 8 sets. Note that he is assured of a large number of documents 
upon which he could perform further logical operations by the fact that 
the term ’’thermal” is posted to 10,954 documents. 

One of the terms displayed in the expansion, "thermal emission," 
seemed, on the basis of the problem statement and his general knowledge of 
the subject area, to be worth examining further. The searcher therefore 
requested a thesaurus expansion from which he selected three terms. He 
then combined these to form set 9. 

To get another start on the problem, the searcher went back to the 
central idea and selected a broad but related term, "temperature," on which 
to request an expansion. He knew from previous experience that asking the 
same question in several ways is often necessary to assure that documents 
dealing with a particular subject (but indexed by different individuals 
using similar but not identical terms) are retrieved. Unfortunately, he 
misspelled it "temperture The display indicated no postings under this 
term. The searcher nevertheless did find what he considered to be a use- 
ful term in this display. He then decided to overcome his spelling defi- 
ciency by using first the root of the term, "tempera.” This produced the 
desired result and he again displayed all the terms up to the limit, 
selecting 12 to form 10 additional sets. 

Next, he chose to perform a thesaurus expansion on one of the terms 
which appeared during the previous display, "temperature measurement. " 

This term, of course, is specifically what the search is about. Since 
the measurement was to be carried out by observing the radiation, he 
chose "radiation pyrometer" as a set. 

Since the search statement had contained the term "radiometer," he 
then decided to see what useful terms might be alphabetically related to 
this. The expansion was reasonably productive but he decided on viewing 
the first display that a thesaurus expansion would be more appropriate. 

From this he selected seven terms. 

He then went back to the general concept of "measurement" and per- 
formed a thesaurus expansion which yielded a number of useful terms. 

In the general sense the concepts of "temperature" and "heat" are 
closely related. Seeking to formulate the idea of "temperature measure- 
ment by detection of radiant heat flux" in other terms, he chose to expand 
the general term "heat." Only three of the terms displayed, he felt, 
added significantly to those he had already selected. 

He then approached the idea of measuring the temperature distribution 
over a surface as a form of mapping and sought to determine whether any 
of the related terms would be useful. From the thesaurus expansion he 
selected only three. 

He then tried a different approach on the idea of measurement by 
using the term "sensing." This appeared to be quite productive. At this 
point he decided to winnow down the number of documents to those more 
pertinent. He combined those sets dealing with temperature measurement 
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to get a single set, number 39. He also combined sets dealing with radi- 
ometers to obtain set 40 and sets dealing with mapping and sensing to obtain 
set 41. The sets dealing with temperature in a general way were inter- 
sected with set 41 to give a new set, 42, which contained 465 documents 
Indexed under both the idea of temperature and that of mapping. The 
general idea of measurement was then combined with set 41. The new set, 
number 43, was intersected with sets on heat and radiation measurement to 
obtain 1443 documents dealing specifically with radiation mapping. Set 43 
was also intersected with sets 39 and 40. The result was a set of 1473 
documents dealing with mapping by radiometer. 

In this process, one term with only two postings caught his eye: 

"sensor for airborne terrain analysis." He decided therefore to display 
the two citations to see how relevant they were. They were not. 

Up to this point he had established that the documents in sets 39, 40, 
42, 44, and 45 with some 9000 postings seemed to bear on the problem of 
measuring or mapping radiation fluxes over large areas. Since he was speci- 
fically Interested in the emission from water he developed some expansions 
about the idea of liquids. Except for the general terms themselves, the 
results did n'ot appear to him to be particularly promising. He then began 
to explore the idea of surfaces and assembled all the documents (495) 
dealing with the Idea of liquid surfaces. 

Backtracking for a moment, he intersected the idea of radiometers 
(set 40) with the concepts of temperature measurement, temperature mapping, 
and radiation mapping (sets 39, 42, and 44). This produced set 53 with 
347 accessions. Ultimately, he decided that set 53 represented his best 
effort on the question, but at the time, he decided to try one more tack 
to see if some useful documents might show up. 

He had not yet investigated the possibilities of approaching the 
question from the point of view of sensing from an aerial platform, which 
is how this job was to be done in the field. He began to travel this path 
by asking for an expansion on airborne. Although he selected one term, 
the expansion did not appear to be too fruitful. He then made a thesaurus 
expansion on the selected term without gaining what he considered to be 
additional insight. At this point he tried an expansion on a general, but 
related, term, "aerial," Again he got only one term which he felt might 
be useful, so that he decided not to continue the expansion. 

The two terms selected from this effort he combined to create set 55 „ 

He was obviously undecided as to what to do with it at the moment because 
he then intersected set 53 with set 52 (dealing with liquid surfaces) and 
got only 4 documents (set 57). This of course is too few, so he then tried 
searching the 2956 accessions in set 56 for index terms also in set 53 
(by performing a logical intersection or "and"). This gave set 58 with 
31 documents . ( 

To see if he were in the right area in his selections, he asked for 
two of the cl'tations from set 57. These looked to be very pertinent, so 
he intersected set 57 with set 58. Since three of the four documents in 
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set 57 also appeared in set 58, he felt that set 58 was probably also quite 
pertinent. He therefore asked that complete bibliographic information on 
the documents in 57 and 58 be printed. Since both were subsets of 53, he 
felt that 53 would probably also contain a large amount of relevant data. 
Finally, then, he asked that all the accession numbers in set 53 be printed. 

As a matter of interest it may be noted that ultimately 388 separate 
relevant documents were found in the system through a total of eight 
searches by different individuals. The particular search discussed here 
found 121 or 31. 2 % of these. Of the 347 hits designated by the searcher 
as constituting his search results, 34. 9 % were relevant. This is very 
good performance for this type of question (i.e., one without highly spe- 
cific indexing). 

Of particular note also is the fact that this search found 55 rele- 
vant documents turned up by no other search. The additional 66 relevant 
documents were found in one or more of the other searches. Only 15 of 
the 388 relevant documents appeared in 5 or more searches. Only 5 rele- 
vant documents appeared in 6 or more of the searches and of these, 3 
appeared in 7 searches. No document appeared in more than 7 searches. 
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SECOND SFARCH LOGIC EMPLOYED BY WWS 

After completing his assigned search, searcher WWS felt that the task 
could probably have been done more simply by employing a single inter- 
section between two groups of very general terms. Accordingly, he began 
by selecting ’'thermal" (12,557 postings), "temperature" (25,518 postings), 
"temperature gradient" and "temperature gradients" (2,428 postings), "maps, 
"map," and "mapping" (4,226 postings). He then intersected the last group 
with the union of the first three to obtain 205 hits. Interestingly 
enough, this search produced only 8 good hits, half of which were not 
found in any other search. 
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LOGICAL PROCESSES EMPLOYED BY SEARCHER MRN 

The searcher is a trained literature chemist with specialization in 
the medical and biological areas. He had relatively low familiarity with 
RECON .and its capabilities at the time the experiment was conducted. In 
sdd i t i on , his thought processes were somewhat disturbed by several short 
but annoying systems fai lures during the course of his search. 

He began (Figure 2) by asking for an expansion on the name of the 
instrument cited in the problem statement as being the one used to measure 
temperature. From this he selected three terms which he combined to form 
set I. It also became obvious from examining this display that a simple 
alphabetical listing of the terms in the neighborhood of "radiometer" 
would not give him a clue as to terms functionally related to radiometers. 

He therefore asked for a thesaurus expansion on "radiometers." This he 
deemed more productive because he combined nine of the 20 related terms 
displayed to create set 2. 

He next tried the idea of pollution, reasoning that because the radi- 
ometer was used to investigate thermal pollution, he might find a connection 
l he dictionary expansion seemed fruitless, so he tried a thesaurus expan- 
siori. Even though he looked at all the terms in this expansion, he decided 
a ^" n on© wa ® real ly suitable. He apparently did not regard the term 
thermal pollution" as being sufficiently descriptive of measurement 
problems in thermal pollution to choose it. 

Following this tack, he examined an expansion about the term "airplane, 
the vehicle on which the sensor is mounted during measurement. Since there* 
are no postings under this term, it was obvious that a different term is 
used by the indexers to describe flying machines. He therefore tried "air- 
craft and found some 10,000 postings. In an attempt to be a little more 
specific, he requested a thesaurus expansion on the term. The phlethora 
of types disp I ayed caused him to settle for the genera! term as set 3. 

He then tried the idea. of "sensor," a concept taken from the problem 
statement. From the expansion he selected only the genera! term for set 4* 
He also tried a thesaurus expansion on "sensors," selecting the general term 
as set 5 and remote sensors" as set 6. To clean things up a bit he com- 
bined sets 4, 5, and 6 in set 7. 

Water , being the medium whose temperature is to be monitored, then 
seemed worth checking out. A dictionary expansion was pertormed with the 
result that three of the terms - "water," "water temperature," and "water 
pollution" - were. selected as sets 8, 9, and 10. On the hunch that some- 
thing very specific might turn up on the intersection of "water pollution" 
and "remote sensors," the operation was performed (set II) and some of the 
citations examined. At that time, the citations did not appear to be 
particularly relevant so the searcher tried another but similar tack com- 
bining sets 2, 4, and 5 dealing with sensors and radiometers to form set 

l2 + 3 ??.! n+erSeC+i " 9 +h ‘ S wi+h " wa T er Pollution" (set 10). The resulting 
set (13) had 36 hits, but an examination of one citation suggested that 
the proper connection had not yet been made. The citation did suggest 
another term which seemed to be related to the problem, "aerial 
reconnaissance" (set 14). 
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An Index term from set II - ’’thermal pollution” - was then used as the 
basis for an expansion without yielding anything of great interest except 
the term itself (set 15). 

The searcher then combined set 6 and 14 to collect documents dealing 
with aerial reconnaissance and remote sensing and searched these for those 
documents also indexed under "water temperature/' "water pollution," and 
"thermal pollution." The resulting set (16) contained 126 documents. 
Apparently he had some second thoughts about the utility of such an inter- 
section because the results were left to lie while he attempted a different 
approach: combining radiometers and sensors (sets 2, 4, and 5) and inter- 
secting these with "water" to create set 17 with 86 hits. This he inter- 
sected with set 3, "aircraft." Only one document was recovered in this 
set (18). Since the number of hits was insufficient, he tried the idea of 
intersecting the water and thermal pollution terms with "aircraft" to 
create set 19 with 142 hits. 

At this point he decided to look to some of the citations in sets 17, 
18, and 19. Apparently set 19 appeared to stray from the subject. The 
others, however, seemed reasonably pertinent. He therefore combined these 
with 16 to create set 20 with 21 i hits. 

His next step was to combine sets 6 and 14 dealing with remote sensors 
and aerial reconnaissance and intersect the result with sets 9, 10, and 15 
which had been used earlier intersected with aircraft to obtain low rele- 
vancy set 19. He seemed to hope that by employing more specific terms 
than aircraft he might obtain more pertinent citations. The new set, 21, 
contained 126 items. Apparently forgetting his history in creating set 17, 
he repeated the procedure to create set 22. He also repeated the creation 
of set 18 to form set 23. Sets 21, 22, and 23 were combined to form set 
24 of 211 items and the results declared the final hit list. This list 
Is substantially the same as set 20. 39 documents in the final hit list 

were regarded by the experiment director as relevant. 9 of these were 
not found by any other searcher. Recall was 10$ of the total relevant 
documents in the file and 18.5$ of the final hit list was relevant. 
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LOGICAL PROCESSES EMPLOYED BY SEARCHER TRP 

Searcher TRP is a graduate electrical engineer with many years experi- 
ence in component and device manufacturing industry. At NC/STRC his 
specialization has been in biomedical equipment. He is familiar with the 
NASA file and with RECON. 

This searcher began his search. Figure 3, by selecting a term associated 
with the centra! thrust of the question, "thermal pollution." He then per- 
formed a dictionary expansion on the term to see if there was related material 
of interest, but found none. He then performed a thesaurus expansion on 
"oceans," and apparently with the idea that since oceans are larger than lakes 
more reports and papers deal with them. Much of the results, however, would 
be applicable in either case. From this expansion he selected 24 terms with 
a total of 4,293 postings to be set 2. 

Following this, he asked for a dictionary expansion on the root 
"thermogr" which relates to the practice of recording temperature measure- 
ments. The expansion suggested to him that he make a thesaurus expansion 
on "thermographs." He decided against using any terms from this expansion, 
however. From a thesaurus expansion on water he selected 7 terms from 
which to create set 3. He then moved on to request a dictionary expansion 
on "remote s-" from which he decided to examine a thesaurus expansion on 
"remote sensors." This, however, did not seem to be particularly useful. 

He next called for a thesaurus expansion on "radiometers" from which 
he selected II terms in order to create set 4 with 3,300 postings. At this 
point he also decided to select "thermograms" as set 5 and "remote sensors" 
as set 6, probably because he felt that the terms he had now seen under 
radiometers were no more pertinent than "thermographs" and "remote sensors." 

He then intersected the union of terms 4 and 6 with the union of terms 1, 

2, 3, and 5. This produced set 7 with 626 hits, a respectable body of 
documents upon which one could probably perform further sorting advanta- 
geously. He attempted this by removing ocean radiometry (the intersection 
of set 4 with the union of sets 2 and 3) from set 7. This left him with 
438 hits (set 9). Examination of the first two citations in set 9 indi- 
cated that this set was not highly responsive to the question. 

The searcher then felt that he should probably attack the problem along 
the lines of "temperature measurement." This expansion was not fruitful so 
he fell back to the base term "temperature." From this expansion he selected 
i4 terms with 43,078 postings as set 10 and another 14 terms with 6,332 
postings as set II. He had some second thoughts, however, about operating 
on such a large group of postings and recreated a display centered about 
"temperature field" from which he selected what he considered to be the most 
pertinent terms (8 with 13,146 postings) in order to form set 13. He then 
intersected this with set 6 (remote sensors) and added to this result the 
items deleted from set 9 dealing with radiometry of oceans (set 8). The 
resulting set of 319 documents he called his final hit list. Of the 319 
hits, 97 or 30.4$ were relevant. These 97 constituted 25$ of the relevant 
documents found by all searches. Fourteen of the 97 relevant hits were 
unique to this search. 
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LOGICAL PROCESSES EMPLOYED BY SEARCHER SRW 

Searcher SRW is a graduate In English Education and is employed by 
NC/STRC in providing search service primarily to the education community 
and in general marketing. He is familiar with search techniques and 
philosophy and, to some extent, with the NASA file. He has no formal 
background, however. 

He began his search, Figure 4, with a thesaurus expansion on mathe- 
matical models. This concept, of course, is related to the interpretation 
of the sensor output. It is a bit strange, however, to begin at this point. 
He continued the expansion to its end without finding any terms he felt 
would be suitable. He then tried the idea of sensors. The thesaurus ex- 
pansion did not seem to him to be particularly fruitful and was terminated 
after the initial display. He then moved to the idea of data reduction. 

The dictionary expansion here did not appear to be useful nor did a sub- 
sequent expansion on "data processing." 

After drawing a blank from this line of questioning, the searcher 
moved to the area of mapping. From the thesaurus expansion, he picked the 
generating term along with "surveys" as set I. A thesaurus expansion on 
"lakes" yielded three terms, "lakes," "surface water," and "thermal 
pollution" from which he formed set 2, Since thermal pollution was pre- 
cisely the thing whose measurement was sought, he also tried a dictionary 
expansion on that term. From this expansion he selected "thermal radiation" 
as set 3. He then combined sets 2 and 3 and intersected this union with 
set I to obtain 26 items. (set 4) dealing with mapping of lakes and thermal 
pollution. 

Searcher SRW then examined the dictionary expansion on "remote sensors," 
selecting only the generating term as set 6. One other expansion occurred 
to him: "temperature measurement." This became set 7. Combining sets I, 

5, 6, and 7, he intersected these with the union of sets 2 and 3. Note 
that this Is an expanded version of set 4. The 172 hits obtained from set 
8 were selected as the final hit list. Forty-eight of these were regarded 
as pertinent by the experiment dlrectpr. Thus while 27.9$ of the hits were 
relevant, only 12.4$ of those identified in the collection as being pertinent 
were recovered. Fourteen of the pertinent hits (or almost 30$) were not re- 
covered by any other search. 
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LOGICAL PROCESSES EMPLOYED BY SEARCHER AWL . 

This searcher Is a graduate chemist with long research experience in 
the textile industry and recent professional experience in textile infor- 
mation retrieval. The NASA file and RECON are relatively unfamiliar to him 

He began his search (Figure 5) with a dictionary expansion on "water 
temperature. The disappointingly (to him) smal i number of postings and 
lack of related terms caused him to terminate the expansion after one 
display and ask for a thesaurus expansion. What he saw then also did not 
appeal to him. It did, however, suggest another avenue of investigation 
(an expansion on "surface temperature"). Before pursuing that, however, 
he decided to determine the possibilities of the term "lake." The die-' 
tionary expansion yielded one set (set I). Then departing from the practice 
of the other searchers, he asked for and printed locally thesaurus expan- 
sions on lakes, surface temperature, radiometer, dicke radiometers, and 
infrared detectors as well as a dictionary expansion of radiometer without 
creating any sets at the time of the individual displays. This fact is not 
evident from an examination of Figure 5, 

From this printed list of terms, he then selected as set 2 "surface 
temperature." He intended to perform an intersection of I and 2 but per- 
formed a union instead. This became set 3. The intersection became set 4 
but had only 3 terms. He then consulted the printout again and selected 
"lakes" as set 5. This he intersected with 2 and got six hits, hardly better 
than " lake"*"surface temperature." Again he consulted the term printout 
and selected "water temperature" as set 7. An intersection of set 5 with 
set 7 gave only one hit, however. These three tries at identifying very 
specific documents he recognized to be failures so he returned to the 
practice of displaying and printing terms. 

A dictionary and a thesaurus expansion on water were first. From 
this he selected "water" as set 9. From the previous printout he selected 
radiometer" as set 10. An intersection of these two items produced set 
II with 49 hits. He then consulted his printout and selected "temperature" 
as set 12. An Intersection of this with set II produced 20 hits. He then 
returned to the printout and selected nine additional sets (14-22) "coastal 
water," "sea water," "great lakes (North America)," "beaches," "coasts," 
ponds, ’reservoirs," "river basins," and "water resources." By this time 
he had realized that if he were going to retrieve a large percentage of the 
pertinent documents in the file he would have to search through a tarqe 
number of documents. 


He continued making thesaurus expansions on water-related terms: 

Jr i vers," "coastal water," "sea water," "great lakes (North America)," 
ponds, and reservoirs." In the process he also created sets 23 24 25 

26, 27, and 28, "rivers," "estuaries," "streams," "oceans," "ponds/ 1 and 
reservoirs.' He also selected "river" (set 29) and "stream" (set 30). 

He then began combining terms putting together 9, 14, 15, |6, 17 |8 19 
20 and 21 to make set 31; and 22, 23, 24, 25, 26, 27, 28, 29, and 30 to' 
make set 32. In set 33 he combined sets 31 and 32. The total postinqs 
in this set were 10,964. 
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. ^ gain he consulted his printout of terms. From this he selected 10 
addit'ona! sets dealing with radiometers. He summed them in set 44 to 
obtain 3,377 postings. He then tried intersecting set 44 with set 33 to 
7'® ld ^ 45 with 151 postings. This set, he expected, would contain 
documents dealing with radiometeric measurements of water. To see what 
output would be generated by another water term ’Make," he intersected 
set 44 with set I and got only I hit. Set 5, "lakes," was also inter- 
sected with set. 44 to give set 47 with 7 hits. He then added sets 45, 

46, and 47 to give a total of 157 hits (set 48). Departing slightly 
from this line of attack, he intersected set 7 (water temperature) with 

/ 4 . 4 cL^ ncl 90t 12 hits * T hese he then added to set 48 to get 164 hits 

set. 50). He then summed set 50 with the results of three previous inter- 
sections: lake*surface temperature, set 4; lakes*surface temperature, set 

8. The resulting set, set 51, had 172 hits. 

Again consulting his term printouts, he selected "measuring instruments" 
as set 52 and radiation measuring instruments" as set 53. Departinq from 
his previous practice, he selected as sets directly without first viewing 
®J<PJ ns | on s several synonyms for that portion of the measuring instrument 
n contact with the phenomenon being observed: sensors (set 54) sensor 

[Hi III’ d ® + f + °^ s i s t* 56) > de+ector (s ®+ 5 7), scanners (set 58), scanner 
(set 59), probes (set 60), and probe (set 61). These 10 he surged to ob- 
tain set 62 with 12,471 postings* 

Having decided that to insure high recall one must examine a larqe 
number of documents, he added to set 33 sets I (lake), 5 (lakes) and 7 
(water temperature) to create set 63. Although he had already used these 
sets previously, he apparently decided to broaden the search somewhat by 
including in the investigation, at least for the time being, possible 
relevant documents which may have been eliminated during previous inter- 
sections. Set 63 had 11,155 postings compared with 10,964 in set 33 alone. 

An intersection of set 62 with set 63 yielded 292 hits. These hits should 
be re lated to measuring things about water, though not specifically the 
radiation emitted. He added these 292 hits to the 172 he had obtained 
previously (set 51) which he feit were reasonably specific with regard to 
radiation measurement of bodies of water, after first ascertaining that 
intersecting either set 51 or set 64 with set 12 (temperature) would give 
a very. small output. He then elected to print the hit list for the 447 
items in set 67. A disc overflow temporary ly interrupted his search for 

the final aspect of the search: mathematical models for interpretinq 

radiometer indications. p a 


When he was able to resume his search, he performed expansions on 
mathematical models," "data reduction," and "computer program." From 
these dictionary expansions he selected "mathematical logic" (set 68) 
mathematical model" (set 69), "mathematical models" (set 72) "data 
reduction" (set 73), "computer program" (set 74), and "computer programs" 
(set 75). The numbers for sets 70 and 71 were unused. The six foregoinq 
items were summed to create set 76 with 34,935 items. This was intersected 
o^'° n °! +herrnal Elation" and "infrared radiation" to yield set 
With 223 postings. After seeing that an intersection of set 80 with the 
term temperature only produced 3 hits, searcher AWL asked that the items 
in set 80 be printed and added to those previously printed (set 67) These 
two groups he called his final hit list. 
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Of this total of 670 documents, 87 or \Z% were judged to be relevant 
by the experiment director. This search found 22.4% of the relevant docu- 
ments identified as being in the NASA collection. Ten of the 87 relevant 
documents were not found by any other search. It may be pointed out that 
although this search utilized a large number of terms and a large number 
postings, comparatively few terms were in the area of temperature 
measurement, radiant heat flux measurement, thermal emission, heat trans- 
fer, etc. This apparently is the reason that this broad search failed to 
retrieve more relevant documents. 
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LOGICAL PROCESSES EMPLOYED BY SEARCHER PJC 

This searcher is an electrical engineer by training. He has had many 
years of industrial research, development, manufacturing, and management 
experience. His professional interests center on devices and systems 
rather than searching specific technical questions. 

( 

He began his search (Figure 6) by requesting an expansion on tempera- 
ture. After reviewing all 41 terms, he constructed two sets, I dealing with 
temperature measurement and 2 dealing with temperature transducers. The 
first display of the expansion on "radiometer" indicated to him that the 
pertinent terms were just the three he used to create set 3 and he did not 
continue to display additional terms. 

Thinking that measurement precision would certainly be a crucial as- 
pect of the search, he selected "precision" as set 4 without first performing 
an expansion. He then returned to the earlier sets and performed an inter- 
section of the union of sets I and 2 with set 3. The resulting set, 5, 
dealing with radiometers for temperature measuring, contained 147 items. 
Fearing that this was too small a group to represent the real content of 
the file, he tried a slightly different tack. He intersected a union of 
the first three sets with set 4. The resulting set, 6, contains documents 
which should deal with the precision with which one can make temperature 
measurements but only partially with radiometers. Only 19 citations were 
so identified. Examination of the full citation for two of these indicated 
that they were not particularly pertinent. Nevertheless, he asked that 
these 19 citations be printed in full and considered part of the final hit 
list. Moving then to the idea of the mathematical models by which the 
radiometer indications were interpreted, searcher PJC asked for an expan- 
sion on "mathematical model" and selected two terms from the first and 
final display to create set 7. He then selected set 8, "temperature 
measurement," and set 9, "temperature distribution," without examining 
an expansion. Performing an intersection of set 7 with the union of sets 
8 and 9, he obtained set 10 with 375 postings dealing with mathematical 
models for temperature measurement. 

He then selected the term "data reduction" (set 11). Backing up for 
the moment, he tried an Intersection of set 7 with set 9 to obtain set I2 P 
a somewhat more restrictive form of set 10 with 298 postings. Documents 
in set 12 should deal with mathematical models for temperature distributions. 
Apparently searcher PJC reasoned that the restricted form contained suffi- 
cient data from which one could sift a suitable number of pertinent hits 
since he used set 12 as the basis of further investigation and dropped set 
10 from further consideration. Before continuing in this vein, however, 
he intersected set II with set ! to obtain 53 citations (set 13) dealing 
with data reduction of temperature measurements. Examination of the first 
three citations revealed them to be of little or moderate interest. When, 
however, an intersection of set 12 with set 4 gave no hits, he decided to 
print set 13 as part of his final hit list. 

Taking one additional tack, he selected "remote sensors" and inter- 
sected this with the union of sets I, 8, and 9, all dealing with tempera- 
ture measurement. This produced set 15 with 107 hits which he chose to 
add to his final hit list. 
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Of the 44 good hits found (out of a total of 179) by this search, 
only two were not found in at least one other search. Despite the limited 
promise of the citations he examined, about 25$ of the hits in the search 
turned out to be relevant documents. Overall recall, however, was fairly 
low. 
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LOGICAL PROCESSES EMPLOYED BY SEARCHER SFS 

Searcher SFS is a young, graduate industrial engineer with a special- 
ization in production but with no industrial experience. He is familiar 
with searching procedures, the NASA data base, and with RECON. 

As seen in Figure 7, he began by attempting a hierarchical (thesaurus) 
expansion on the basic instrument used to make the measurement of lake 
temperature from an overflying aircraft (radiometer). Since the term did 
not appear in the singular in the dictionary, he asked for an expansion of 
that portion of the alphabetical dictionary. In this fashion he found 
that thesaurus expansions are generally made about plural forms. He selected 
radiometers as a set and then went on to perform the expansion he had origi- 
nally set out to do, i.e.,on rad iometers . From the resulting display of 
20 terms he grouped eight to create set 2. 

Since the customer had been concerned about possible accuracy of 
measurement, searcher SFS next asked for an expansion on the word "accuracy." 
Other than the term itself (set 3), nothing else in the display appeared to 
be of interest. He then tried a thesaurus expansion on "accuracy." From 
these displays he selected three terms from which he created set 4. He 
then intersected a union of set i and 2 (radiometers) with a union of sets 
3 and 4 (accuracy). This produced 132 hits. Upon display of the first 
three citations, he decided that the index term "calibrating" was respon- 
sible for retrieving these citations which were not directly pertinent. He 
therefore dropped calibrating from set 4 thereby creating set 6. He then 
repeated the intersection (this time with set 6 instead of set 4) to obtain 
set 7 with only 20 hits. The first three citations from set 7 were only 
slightly more pertinent than those from set 5. This indicated to him that 
if he were to find a significant number of pertinent documents, a new tack 
would be required. 

He tried an expansion on sensor technology but was disappointed with 
the neighboring terms and therefore discontinued the expansion. He then 
decided to perform a thesaurus expansion on "technology" with the result 
that he learned the term does not appear in the thesaurus. A dictionary 
expansion on the same term showed it did appear in the pre-1968 file with 
2,556 postings and that "technologies" was the correct term for the post- 
1967 fife. From these two terms, he created set 8. This he then inter- 
sected with set 2 (the thesaurus expansion on radiometers) and obtained 
only two hits which he displayed. They were not pertinent so he decided 
to accept as his final hit list the 132 items from set 5, knowing of course 
that the relevance of at least the first three citations was not good. 

He later redid the search along essentially the same lines and got a total 
of 160 hits. The latter items were the ones evaluated. Some 60 items 
from this group were regarded by the experiment director as pertinent. 
Thirty-one of these were not found by any other searcher. The 60 perti- 
nent items found In this search can be compared with the total number of 
pertinent items found in all searches (388) to point out that while the 
relevance is high, the recall is only 15.5$. 
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LOGICAL PROCESSES EMPLOYED BY SEARCHER CLN 

This searcher is an aerospace engineer with a research and develop- 
ment background (primari ly in structures) and a familiarity with the data 
base and with RECON. 

He began his search (Figure 8) by requesting an expansion on "radi- 
ometer. " Although he only looked at the initial display, he selected four 
terms centering around "radiometer" from which to create set I. He then 
requested a thesaurus expansion on "radiometers” selecting 9 from which he 
created set 2. 

His next idea was to examine the potential of terms related to thermal 
pollution. The dictionary expansion, he concluded, was not very productive 
so he tried thesaurus expansion. From this he took 13 terms to form set 3. 
One of the other terms in the expansion, "Lakes," seemed to him to offer 
an opportunity to examine a very pertinent aspect of the problem. He 
collected 9 terms from the thesaurus expansion to form set 4. Then, 
approaching the crux of the matter, he requested an expansion on "water." 

Set 5, taken from this expansion, deals with "water," "water intake(s)," 
and "water management" and has some 8,700 postings. 

He then approached the idea of "remote sensing" but when the term was 
not found in the thesaurus dictionary, he abandoned the idea and proceeded 
to intersect the union of sets I and 2 with the union of sets 3, 4, and 5. 

The resulting set, 7, should contain documents dealing with radiometry of 
water systems. Set 7 consists of 230 items. The searcher displayed the 
accession numbers of these and then displayed the full citation for a new 
N-document and an older A-document. These appeared to be approaching the 
core of the matter; in particular, the first item seemed quite pertinent. 

The searcher next left these documents and proceeded to examine another 
aspect of the problem statement; a mathematical model of the emission, 
transmission, and reception of radiant energy. He began by requesting an 
expansion on "mathematical model" and selected two terms from among the 24 
that he examined. Out of these he created set 8. Since the customer, as 
indicated by the problem statement, was interested in solving any mathemati- 
cal model deemed appropriate by computer, the searcher next asked for an 
expansion on "computer! zed . " The first display convinced him that in asking 
for an expansion on "computerized," he had passed the point in the 
dictionary where the terms of most interest lay. He therefore asked for 
an expansion on "computer" and from this selected four terms with total 
postings of 16,367 as set 9, 

Continuing with the idea of computerized processing of data according 
to a given mathematical model, the searcher asked for an expansion on "data." 
He formed set 10 with 12,745 postings from 5 of the terms in this expansion 
which, incidently, he carried through to 40 terms. Next, he asked for a 
thesaurus expansion on "Data reduction." The results he did not care for. 

He then intersected a union of sets 8, 9, and 10 with a union of sets I and 
2 to obtain set ii of 288 postings which dealt with modeling and processing 
of radiometer data. Apparently, the number of documents in this set was 
somewhat larger than he reasoned might be really pertinent so he performed 
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(1+2) * (8+10) and got set 12 with 256 hits; (1+2) * 10 and got 202; (1+2) * 
(8+9) and got 102. Each of these subsets of II were fairly large, indicating 
substantial overlap in indexing. He tried to get somewhat more specific hits 
by performing (1+2) * (8+9) * 10 which yielded set 15 with 16 hits. 

Backtracking, the searcher then combined sets 7 and II to obtain set 16 
with 504 documents dealing generally with radiometry, water systems and data 
analysis. At this point, he decided he would go back to the thesaurus ex- 
pansion of "radiometers." From this expansion, he selected five terms, 
instead of the nine he used to form set 2, to create set 17. He then per- 
formed an operation similar to the creation of set II, except that he re- 
placed set 2 by set 17. This gave him 164 hits (compared with 288 in set 
II). He added to this set, set 18, the 230 documents from set 7 dealing 
with radiometry and water systems to form set 19 with 387 documents. This 
he called his final hit list. Ninety-seven of these hits were considered 
good (relevant) by the project director. Eighteen of these were not found 
by any other searcher. 
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CORRELATION OF SEARCH RESULTS WITH SEARCHER’S BACKGROUND 

Results produced by the eight significant searches on this subject can 
be divided roughly into two groups: those which gave at least 87 good hits 

(average 101) and those with 60 or fewer good hits (average 48). The first 
group of searches was written either by experienced searchers or by engineers 
with a working knowledge of fhe subject area. Searcher AWL^compensated for 
his lack of familiarity with the subject area by doing a rather broad search. 
His recall was fairly good (22. 4 % compared with 31.2$ "t-he best se ®[ ch 
in the group) but his relevance was low C 1 3% compared with 36.6% for the 
best in the group). Searchers TRP and CLN, the most experienced in the 
group on the NASA file, both have backgrounds in areas related to the sub- 
ject area but not directly on it. Their searches gave good reca 1 1 and re I e- 
vance and, in addition, employed considerably simpler logics than the batch 
search or the one done by WWS. The latter two searches were done by working 
engineers with backgrounds fairly close to the subject area. From the! r 
results it seems that being more diligent usually will lead to a slight 
improvement in recall percentage. 

The other four searches were performed either by searchers with no 
relevant education or experience or by searchers with little regular search 
experience. Searcher SFS, closest of the four to the subject area by back- 
ground and search experience, also did the best job in terms of relevance 
and recall. While his search had the best retevance of any search done on 
this subject, his unfamiliarity with the subject undoubtedly contributed 
to his failure to follow synonymous paths to the other documents on the 
subject and thus limited his recall. 
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CORRELATION OF SEARCH LOGIC WITH RECALL ACHIEVED 

As the Initial investigation into the effects of search logic on the 
recall achieved, the reasons searcher WWS failed to retrieve some^of the 
documents found in the batch search were studied. It was found that 
excluded documents were generally indexed under one of three groups of 

terms : 

(1) INFRARED INSTRUMENTS 
INFRARED RADIATION 
INFRARED IMAGERY 

(2) THERMAL POLLUTION 

(3) SURFACE TEMPERATURE 

It was then found that the terms in group (I > do not appear in the thesaurus 
expansion of radiometers. Thus, unless the searcher happened to thmK along 
these lines, the terms would not have been suggested to him otherwise. A 
similar situation exists with respect to group (3). This term does not 
appear under a thesaurus expansion of "temperature. Failure to 'nclude 
"thermal pollution" seems to have been just a mental lapse on the part o 

searcher WWS.. 

No comparisons with the batch search are possible for P re ~ | 968 docu- 
ments because the batch search excluded these. Thirty-six good hits were 
found by the WWS "A" search for this period. 

The WWS "A" search found 58 good hits not found by the 152-term batch 
search; only 27 good hits were the same in both searches. The reason, 
apparently, is that the batch search employed a doub le. i ntersection along 
the following lines; (temperature measurement) * (radiometers) . (accuracy). 
Examination of 24 of the good documents found by WWS but missed in the batch 
search indicated that these documents were not indexed under a term synon 
ymous with accuracy such as calibrating, correction digital ® +c *’ 

all of which were used in the third group of the intersection. They were, 
therefore, not selected. 

In general one must conclude that, for this topic at least, high recall 
can only be obtained by; 

1. employing every conceivable term related to the subj'ect 

2. single intersections only 

3. extensive manual sort down. 

One may also conclude that while each of the searches employed a valid search 
loqic those that employed the most terms and the most different approaches 
to 9 !^ same topic were the most successful in terms of recall. Some approaches 
of course favored higher relevance than others. Searches employing These 
approaches only appeared to be highly (y30%) relevant but were so at the 

expense of recal I. 
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RELATIVE EFFECTIVENESS OF INTERACTIVE AND BATCH SEARCH MODES 

A second experiment, this one attempting to access the relative 
effectiveness of the Interactive and batch search modes, was also con- 
ducted. Engineering time to conduct the search was compared with search 
results as measured by good hits. As shown In Table 2, engineer CLN got 
essentially the same results (using substantially the same logic) by 
RECON as he did by the batch process. The search time was almost iden- 
tical. On another question it took him somewhat longer to search via 
RECON (probably because of long question response times). Again the good 
hits were substantially the same Items. 

On question #4, PJC took 25# more time to do the problem on RECON as 
by the batch process, while searcher SFS took only half as long to get a 
few more good hits on question #5. In the latter case, he spent a good 
deal of time working out the strategy prior to the batch search. When he 
went to the RECON terminal, he knew generally how he was going to attack 
the problem. 

Generally, the effectiveness of a RECON search was not better than 
that of a batch search conducted by the same individual using the same 
amount of engineering time to prepare the search. One can also corre- 
late reasonably well, using a logarithmic function, the engineering time 
spent in preparing the search with results (recall) achieved. 

One final piece of data bearing on this question is the study made 
of searcher TRP's batch and RECON searches during a given period (Tables 
3 and 4). Some )5 batch searches and 10 RECON searches were studied. It 
was found that his RECON searches achieved about twice the average rele- 
vance of his batch searches at the cost of about 35# more engineering time 
per search. 
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RELATIVE COSTS OF BATCH AND INTERACTIVE SEARCH MODES 
Cost Structure for the Batch Retrieval System - STRC-IVS 
Di rect Costs 

Engi neeri ng Time 

This is the time spent by an engineer (or search analyst) In 
preparing a search strategy for input to the computer and in eval- 
uating the search output once his strategy has been run. Typically, 
this time is spent in several ways: discussing the search problem 

with a client, looking through the appropriate thesaurus for terms, 
checking those terms against a computerrproduced list of authorized 
terms and postings, composing the search strategy, and then finally 
reading or scanning the hit abstracts for relevant documents. STRC 
cost accounting reports show that, on the average, an engineer spends 
I 1/2-2 hours In preparing a search and 2-2 1/2 hours In eval- 
uating the search output at a total cost of roughly $35 to $40. 

Cleri cal Time 

Clerical time for a search includes such functions as inter- 
preting search requests, verifying search terms and postings, key- 
punching or keyboarding via terminal the actual search request, 
cheteklng search output, retrieving and Xeroxing hit abstracts, and 
binding search bibliographies. Accounting statistics show that on 
the average, the clerical time (two hours) involved in a search 
costs STRC $5 to $6. 

Xeroxing Cost 

The average cost for Xeroxing of search output is roughly $7. 
Computer Cost 

STRC searches run on an overnight, no-priority basis on the 
Triangle Universities Computation Center’s (TUCC) IBM 370/165. The 
TUCC charges for zero priority service include the following: 

1) CPU time 8 $350/hour 

2) EXCP (I/O) time § $262. 50/hour 

3) Core use % $.70/ki lo-byte hour 

4) Unit record I/O transmission (i.e,, printing time) 

§ $. 00085/UR (line) 

Accounting records show that on the average, the computer cost for a 
NASA search ranges from $10 to $20. 

The total direct cost for an STRC batch search then averages $60 - $70. 



Overhead for the NASA Fi te 

A number of overhead items are incurred in maintaining the NASA file 
and the costs involved are absorbed by STRC. 

Monthly File Maintenance 

1 ) $50 to $75 computer cost 

2) $5 clerical cost 

Di rect Access Storage 

!) Rental of five 2314 disk packs @ $9. 50/pack/month 

Magnetic Tapes (Back-up and Maintenance) 

I) Approximately 50 tapes at one-time purchase price of 
$9. 50/tape. 


Discussion of Possible Cost Structures for RECON Interactive Searches 

Since STRC is not aware of any cost accounting scheme that might exist 
for RECON searches, one can only speculate on what the elements of such a 
scheme might be. It is logical to conclude that one or more of the following 
items might be included in the cost/search for RECON. 

Computer Cost 

Costs for interactive services vary from machine to machine, but 
it may help to give an example of the TUCC charges for interactive 
service on the 370/165: 

1) CPU time @ $1000 /hour 

2) 1/0 channel program time % $750/hour 

3) 1/0 wait time % $375/hour 

4) Port connect time @ $2/hour 

These costs are definitely high estimates for REC0N since it runs on 
a less expensive machine, the IBM 360/50. The RECON charges would 
probably run less than 1/5 the TUCC charges and maybe as iow as 1/10 
for CPU time. 

Communi cations Cost 

Thfs would include terminals, communications controllers, leased 
lines or dial-up ports, and modems. One example of cost in this area 
Is that of the leased communications line to College Park @ $150 - 
$200/month. This cost would be reduced if STRC were to install dial- 
up REC0N service ® roughly $. 28/minute. The Bunker Ramo terminal and 
controller now in use with RECON leases for $345/mon+h. 
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On-Line Storage Cost 

It is estimated that a fi le the size of the NASA RECON file(s) 
could cost as little as $800/month for on-line direct-access storage 
depending on the devices used. 

Peop I e Cost 

Systems and Programming Support 

This includes people time at NASA and STRC in monitoring 
the system. 

Engineering or Search Analyst Time 

Here the engineering functions would be essentially the 
same as in batch but the amount of time involved might be quite 
different. The engineering functions of preparing a search would 
be performed at the terminal, and the time involved would depend 
quite heavi ly on the response-time of the terminal and on the 
amount of browsing that the engineer might engage in. Average 
times for this activity on experimental searches are two to 
three hours. In addition, the engineer might carry out some of 
his search evaluation at the terminal and thus reduce the amount 
that he normally would have to perform at his desk later. Cost 
for engineering time would be at least equal to the $35-$40 cost 
for batch engineering time and may even exceed it. 

Clerical Cost 

Obviously, with RECON, the clerical functions or preparing 
a search for entry Into the computer would be eliminated. The 
only clerical time involved then would be the time spent in 
pulling and Xeroxing abstracts and in binding search results. 

This averages approximately $3-$4/search. 

A more useful rule of thumb for the possible pricing of RECON searches 
would be to use a charge structure similar to that of one of the organizations 
(such as Lockheed or Systems Development Corporation) that offer nation- 
wide interactive on-line search service on a number of files. This approxi- 
mate charge would substitute for all of the above cost elements with the 
exception of the people cost. These organizations normally charge the on- 
line user a fixed hourly rate for his terminal connect time (i.e., the total 
elapsed time in the system from sign-on to sign-off). These hourly rates 
range from $25-$80 per file searched. For a file the size and complexity 
of the NASA file, the rate probably would be in the neighborhood of $50/hour 
for file access. In addition, the user must pay the telephone company for 
direct-dial service (or use of the TYMSHARE network) to the computer. From 
STRC to College Park, the phone cost would average $)5/hour. Therefore, 
our hourly rate (based on a Lockheed-type pricing scheme) for RECON service 
might be $65/hour. 
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Batch vs. RECON Cost Comparison Based on Estimated RECON Cost 

If one multiplies the Lockheed-type hourly rate by the two to three 
average hours per search that were spent at the RECON terminal during 
experimental searches (see TABLE 2), then he comes out with a computer cost 
figure ($130-$ l95)roughly 10 times that of the STRC average batch cost 
($!0-$20). Since experimental results show that the engineer does not spend 
less time in preparing a RECON search and may likely spend more, the cost 
for his time will be at least what it is for a batch search ($35-$40). 
Although there may be some savings in the clerical cost, they are miniscule 
($2-$3) in comparison with computer and engineering costs and Xeroxing 
costs should remain the same. The total estimated cost for a RECON search 
($175-1245) is then roughly two or three times that of a batch STRC search 
<$60-$70). 

it also should be mentioned that the Lockheed-type hourly rate that, 
was used in estimating the RECON costs may be unrealistically low as it is 
based on a rather high volume of usage and it may exclude some very high 
overhead costs. 


CONCLUSIONS 

Although this discussion has been attempting to compare apples and 
oranges and the Author is fully aware of that, the evidence (based on 
similar on-line systems) at this time seems to show that the batch approach 
to searching NASA is two to three times more economical then the RECON 
approach. However, no attempt has been made in this section to judge the 
relative values of the results of each approach, and that has to be an 
important consideration in the total picture. 
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SUBJECTIVE USER EXPERIENCE 


CThe thoughts presented below were complied by NC/STRC’s most experi- 
enced RECON user. While they are those of a single individual, they do 
seem to parallel rather closely the experiences of other RECON users on 
the staff at comparable points in their acquaintence with RECON. 3 


One of the first problems that faces a novice at the RECON terminal 
(if a non-typist) is that of the keyboard. Most of the difficulty dis- 
appears after a few hours of practice, but the problem of where the "F" 

(or some other elusive letter) went remains to plague the user for many 
months. This produces a tendency to use EXPAND functions for no other 
purpose than to select a term which appears to be too much trouble to 
type. I suspect that even an accomplished typist might be tempted to do. 
the same, particularly when the desired term is an especially long one or 
when the exact spelling (singular or plural, hyphen or space, etc.) is not 
certain. 

The big problem, which although purely psychological is not unimpor- 
tant, is the nagging suspicion that the fumbling approach to initial use 
will result In the grinding halt of a multi-million dollar computer prob- 
ably accompanied by sparks and smoke at the other end of the phone line. 
Initial instruction to the operator should make clear that nothing could 
be done either to harm the system or even to produce a ton or two of 
computer printout. The worst that can happen is that the operater is in- 
formed, "I FOOL, WRONG...” 

The uncertainty of what will happen under given circumstances leads 
the novice to make much more use of the EXPLAIN function than Is ealled 
for. Although much of the material under command of the EXPLAIN key is 
basic to operation of RECON, it appears to be on one hand overly simplified 
and on the other hand too sophisticated for other than academic interest. 
Basic information concerning key functions appears to be dwelt upon with 
excruciating detail while practical material of use to the operator is 
difficult to find. As examples, the EXPAND and COMBINE functions are pur- 
sued in great detail. These functions are normally well understood before 
one appreaches the keyboard. The error codes which are normally disregarded 
with a "try-again-maybe-if I l-work-this-time" attitude take up a large part 
of the EXPLAIN file. In practice, I doubt that anyone ever looks at error 
codes other than as a matter of curiosity. On the other hand, how does one 
determine the techniques for obtaining material from the ’’Corporate Source" 
file? Or how does one know if selections made with a range function are 
inclusive or exclusive of the cut-off indicated by the printer? Academic 
information available from the EXPLAIN function appears to completely swamp 
or obscure information that is of primary importance to the user. 

Much of the above criticism could be remedied by the availability of 
a detailed user's manual which explains the operation of RECON from the 
beginner's standpoint. Although an in-house manual was prepared for the 
use of novl ce • operators at this location, I feel that a professionally pre- 
pared operator's manual is a dire need. Many hours of trial and error at 



the console could be spared If a thorough, concise operational handbook were 
available to the novice searcher. Although the text should be primari !y 
concerned with operational matters. It should include systems information 
to enable a searcher to appreciate why it may take a minute or more in 
order to turn a page. 

When a search is undertaken, the EXPAND key Is probably used more than 
all others combined, I normally approach the terminal with a tentative 
strategy in hand which includes all proposed terms and a Boolean expression 
which combines them. These are obtained by use of a printed Thesaurus and 
postings lists and by a manual search of annual, cumulative indices of the 
abstract journals. The provisional selection of terms from printed sources, 
rather than using the EXPAND functions to obtain them, is a big time saver. ^ 
i estimate that terms can be found up to 10 times faster. Often this material 
is reinforced with a list of known ’’good" accessions numbers which have been 
obtained from a quick manual search. The EXPAND function serves its primary 
purpose as a mean toward selecting terms, either in the alphabetic subject 
listing or the related subject "Thesaurus" listing, which have a close "group 11 
relationship. For example, a group of terms which have a subject heading of 
"LASER " or a hierarchical heading such as "SEMICONDUCTOR DEVICES" can be 
se I ected on large collections from expansions of the SUBJECT or THESAURUS 
files. This ability to quickly select multi-term sets Is one of the most 
appreciated abilities of RECON. 

The SELECT key is the second most used key. It is probably used more 
often than It should be. There is a decided tendency to select ranges of 
terms, both from the subject and Thesaurus listings which include non- 
destred or irrelevant terms, simply because it is easier to do so than to 
exclude those which do not apply to the question at hand. This sometimes 
includes an overabundance of material in the final result which was not 
desired and appeared only because an "uncalled for" term (or terms) was 
unwittingly included. Since the Thesaurus postings provide an easy access 
to related terms, there is also a tendency to favor that portion of the file. 
This may or may not be desirable since that portion of the collection is 
the most recent; but I sometimes feel that the pre-1968 material gets 
short shrift (because term selection is in general more difficult) since 
there is no easy way to obtain a selection of hierarchical related terms. 

After a collective selection has been made, using a range of terms, 
all terms having the same root, or other devices to select a collection of 
terms, it is fairly difficult to determine what has been done after the 
fact. Under the best of circumstances, one is left with a series of num- 
bers which must be compared with an expaasion on the terminal in order to 
determine the exact terms included. Under circumstances involving long 
strings of selected term codes (El, E3, E4, E6, E7, E9, Eli, ...) the 
printout display does not include the end of the series and leads one to 
wonder just what terms were chosen. 

The COMBINE key, which brings a search to its fruition, is a real joy 
to operate. The ability to add, ddlete, and otherwise manipulate selected 
sets in real time is a boon to the searcher. It completely eliminates the 
problem of "should a given term be included because of its inclusiveness or 
excluded because of its generalness!" The ability to use or not use sets 
in order to arrive at a manageable result is one of the major satisfactions 
of RECON. 
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The DISPLAY function is another major advantage of RECON. The ability 
to observe results immediately is extremely valuable to the operator. B v 
perusing the results of a strategy, one quickly becomes aware of the strenqths 
and weaknesses of the strategy as used. This provides an opportunity to 
have second thoughts about the propriety of using certain terms or of com- 
bining them in a given fashion. Often overlooked terms are uncovered and 

he unwise use of other terms is apparent,. I very rarely leave a search 
done on RELON in its original form. I very rarely rerun a search done on 
— batch system. The ability to specify display form with the FOFWAf key“i s 
a time saving device when checking search results. By using only the "of 
interest material available in a citation, normally title, notation of 

content and terms, it is possible to save many page turnings and therefore 
many minutes. 

Most of the other function keys require no comment. They are needed 
and do their jobs with dispatch. It is often difficult to be patient and 
understanding when it takes longer than a minute to turn a page, but the 
page eventual ly turns. It is also a soul soothing satisfaction to be able 
to check the status of the most recent command. It is much better to know 

+hnL°i! I S nu ^ er f i v L' n I he wai+5n 9 line than to simply hope that some- 
thing is happening at the other end of the line. It is rather like blow- 
ing in a silent telephone to see if sidetone is still there. The TRANSMIT 
Key has more frustration built in than I feel necessary* After it Is 
pressed, it is too late to make any change, correction, or bail out until 
the terminal is polled. When the system is busy, there is often more than 
enough time to abort a command between the transmit command and a poll. 

It should be possible to do so. F 

. M y s+ erious messages appear at irregular intervals on the hard copy 
printer. These messages are often (it only appears to be "usually") in- 
complete or garbled. While this printout, along with the information dis- 
played on the video screen, sometimes indicates the status of the system, 

feel that it is a poor excuse for real information about what is really 
going on. if it were possible to query the system to determine status at 
any given moment,^ it would lend a much more satisfying atmosphere to the 
operator s sensibilities. It would certainly help to determine which of 
the numerous possibilities (terminal broken, printer inoperative, phone 
line down, Washington bombed, etc.) keep things from happening in the 
proper manner. In particular, there is a need to know why processing of 
a command sometimes takes so long ('-*40 minutes). 

. * n summ 3Tion, the RECON user can find many things that he feels can 
be improved. Most of them will boil down to a matter of personal preference 
or a matter of how long a waiting period is required before someth?ng 
happens. From a standpoint of effectiveness as compared to batch or off- 
line searching I feel that there is no comparison. The RECON is much more 
effective, and I personally would not consider an off-line search if RECON 
were available. Since I normally do the same amount of preparatory work 
° r search, there are no economies to be gained before the 

search is actively pursued. I suspect that manual preparation and direct 
ey oar input about break even. The ability to moni tor-as-you-go, however 

dS£m S COn T! n ? t,me aS one a++em P +s +o optimize results. In the end, a ’ 
RECON search takes more time than one done off-line. At the same time, 

f ® e * ver V strongly that the RECON search does a much better job as far 
as^both retrieval and relevance are concerned, I wouldn’t do it aay other 
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APPENDIX 




TABLE i 

RECC8 PROBLEM NUMBER 1: RECALL AND RELEVANCE REPORT 


HIT ACCESSION 

numbers 

PJC 

AWL 

CLN 

MSN 

TBP 

SFS 

FOS 


WWS2 

SBH 

A6 9-31543 





X 






A69— 33280 







X 




A69- 33286 

X 

X 

X 


X 


X 

0 



A69-36071 






X 





A6 9- 42 2 1 1 








X 


C 

A70-14617 







X 




A70-15770 


X 

X 

X 




X 


0 

A70- 16363 








X 



A70- 17104 







X 




A70- 17415 







X 




A70-19616 






X 





A70-22014 



X 








A70-22869 

X 

X 



• .0 



X 



A70-22 885 


X 

X 


X 






A70-23525 






X 





A70-23536 

X 




X 


X 




A70-23754 







X 

0 



A7 0-23756 

X 

X 



X 


X 

G 



A70-23763 










X 

A70-2474C 



0 

0 

0 



X 



AT 0-24741 




X 







A7 0—2 5 6 36 

. 


0 


X 






A70- 2650? 





X 






A70-269C3 

X 




0 


X 

0 



A7Q-26 905 

X 




0 






A70-26949 

0 




0 



X 



A70-27689 








X 



A7 0-29655 







X 




A70-29658 


1 





i X 




A7 0-29659 











A70-31231 

X 




X 



0 



A7 0-31806 






X 





A70-33170 

X 




X 


X 

0 



A70-339C5 


X 









A70-35617 










X 

A70-35621 

X 






X 



0 

A70-36967 


0 

0 

X 







A70-37087 






X 





A70-39147 






X 





A70-39798 






X 





A70-408C3 

0 




X 






A70-43C87 






x 





A7 0 - 4 4 6 4 8 

0 


0 


0 



X 



A70- 45363 







X 




A71-10135 








X 



A71-16672 








X 



A7 1-21270 








X 



A7 1-23 217 



X 








A71-2359C 





X 


X 

0 



A71-24685 








X 




A-l^ 


































fi*CC» FBCBLEH NUMBEK 1: EECA-LL AMD 'ttBPOKi 


HIT AC CSS SION 
NUMB SB S 


16 3^.34385 

164-1 1916 
A64-1 3054 
164-16324 
A54- 18229 

164- 27107 

165- 12016 
16 5- 1 45CS 
165—1 6 4 1C 
165- 17 643 
A6 5— 2 3 621 
165*24582 
S. 6 5.-2 9 886 

167-10274 
157-27606 
A67— 31 339 

167-340C1 

167-36521 
A67-40353 

167- 41373 

A67-41789. . 

168- 15566 

168-1 5646 

168- 19734 
A6 8-2 0370 
A68-20492 
A6 8- 21 362 
A6 8-2 4736 
A 6 8-2 8 257 

168-32124 
16 8-32593 
168-33895 
168-34972 
A68-37S54 
A6 8-4 0 9 88 
16 8-4 2 874 
A 68— 43279 

168- 43280 . 

159-1 0693 
A69-12435 

169- 12457 
A6 9-13933 

16 9- 14653 
159-20455 
A6 9-22166 

169-26976 
A69-26997 

169-29515 
169-29528 
169-29534 



A~2 


O X 




















HBCON PROBLEM NUMBER 1: RECALL AND RELEVANCE REPORT 


HIT ACCESSION 
NUMBERS 

PJC 

AML 

CLN 


TfiP 

SFS 

res 

NWS 

NWS 2 

A71-26174 






X 

X 

0 


A7 1-27 260 





X 





A71-30126 







X 



A71-30916 






0 

X 



A71-322S6 








X 


A71-36364 

X 




X 



X 


A7 1-40224 






X 




A71-41300 










A71-41986 

X 

0 

X 


X 


X 

0 


A71-42126 








X 


A71-42150 










A 7 1-42151 







X 



A7 1-43512 






X 

X 



A72-10546 

0 

0 

X 


0 


X 

0 


A72- 11733 



X 







A72-1 1 766 

0 




0 


X 

0 


A72-117S5 





X 





A7 2-1 1 803 

X 




X 


X 



A72-118C4 






X 


X 


A72-1 1808 







X 



A72-1 1810 


0 



0 


0 

X 


A72-11866 


X 



X 



X 


&7 2-1 1 877 


X 





X 



A72-1 1866 






0 




A72— 1 3441 






X 

X 



A72-15909 








. 


A72-16599 


0 

0 

X 

0 




1 

A72-16600 



0 

X 

0 





A72- 16912 







X 



A 72-1 7385 



0 

X 

0 





A72-17430 





X 





*72-17925 



X 



X 




A72-20692 








X 


A72-21824 








X 


A72-22652 






X 




A72-233C1 



X 







A72-24749 


0 



X 





A72-2 5348 







X 



&72-26C83 



0 


0 


X 



A72-277C6 








X 


A72-30587 








X 


A7 2- 3 1 2 4 2 



0 


X 





A72-31252 




X 






A7 2-33630 










A 7 2- 3 3 6 3 1 








X 


A7 2-3 5495 








X 


A72-37C05 






0 

X 



A72-37196 








X 


A72-37252 




, 


X 




A72-39640 






s_L 




-3 


K C ■ X I o X O * I X o I x X 


























RECCB PROBLEM HU MB B H T: RECALL AND RE, LEV AHCE REPORT 


HIT ACCESSION 
BOMBERS 

PJC 

AtfL. 

CRN 

R£N 

T.Itp 

iiFS 

F.OS 

BBS 

WHS. 2 

S R & { 












£72-39677 

0 




0 


X 




A7 2- 3 9 7 95 
A7 2- 4 4 274 
£72-45107 

0 


X 



X 


i 


0 

A 73- 10590 
A73-10833 






X 

x. 

X 



A73-12361 
A73-1 5773 

X 




X 

X 





A73-15780 
ft? 3- 1 8317 
A73-20673 
A73-21 622 






X 

0 

X 

X 

X 

X 



ft? 3—22510 
A7 3-23634 
A7 3—2 5 774 

0 


X 

0 

0 


X 

X 



473-26252 
ft? 3- 26 30 5 






X 

0 

X 



A73-26514 
A73-2 8020 
£73-33360 

X. 



X 

X 

X 


X 

■ 

0 

$62-12628 


X 

O' 








W63-20195 


c? 

•' ' X- 








W6 3-207 10 


X 









$64- 10 915 
B64-1 1474 


f , 0 

X 

1 




X 



N64- 13743 
$64-16705 


■ 

X 

. 




X 



N64-19254 
$64- 19406 
$64—19629 


X 

. X 



: 


1 

X 

X 

• 

i ; 

B64-207C4 

f 







X 



$64-23939 

1 


. 





X 



$65-10573 

$65-21425 








X 

X 


$65-23528 

$65-27649 


X 






X 



$65-30376 


X 

0- 








$65-30377 


' X 

X 








$65-30756 
$5 5-31 359 
$65-32220 
*55-33583 . 


0 

X 

X 




X 

X 


$6 5- 335SC 
165-33597 
M65-335S8 


: X 

X 





X 

X 


$65-33599 


X' 


X 





X 


$65-33602 


X 

0 





X 



$65-35196 

$65-35518 


X 

X 





X 

\ 


’£6-12619 



X 


‘ 





, . .J s 



A -4 




1ECCN PEOBLIB NUMBER 1: RECALL AND RELEVANCE REPOET 


HIT ACCESSION 
NUMBERS 

PJC 

A UL 

CLN 

KEN 

Trip 

m 

EOS 

If 

WHS 

WWS2 


N66-13422 









X 


N66— 2 1223 








X 



N66-22546 


X 









N6 6—2 3 806 



X 








N66-28422 


X 

X 





X 



N66-31 370 








X 



N66-33 394 



X 








N66-36761 








X 



Nb 7— 1 3 466 


0 


X 







N67- 13492 


0 

0 

X 







N67-13511 









X 


N67-1 3525 


X 

X 








N6 7-1 5350 


X 

X 





X 



N67-19838 


0 

X 








N 6 7- 2 0 7 2 8 








X 

X 


N67-24302 








X 



N67-28S46 


X 

X 

X 




X 



N67-29052 








X 



N67-30471 


0 

X 








N6 7-3 0 735 


X 









N6 7-34744 








X 



N6 7-35 89U 








X 



N68- 1 0 677 











N68-1 4108 


X 





MM 




N6 8-14967 


X 



X 


KH 



0 

N68-16117 






■ 





N68-19309 






mm 

■fl 




N6 8 - 1 9 3 3 2 







1 li 




H68-1 9465 


X 









N68-19845 











N68- 1 9870 





X 






N6 8-2 0041 


X 



X 






N68-20264 







X 




N68-21 7 36 






X 

X 




N6 8-2 3 1 2 1 



X 








N68-23710 








X 



N68-24292 






X 





N68-27972 


X 









N6 8—2 8 922 


0 



X 






N6 8-300 12 






X 

X 




N68-304SS 


1 

c 

X 




X 


0 

N6 8-30637 






X 





N68-30647 


X 

0 

X 

X 






N68— 33 396 





X 






N68-33400 


II 


X 




0 


0 

N68-334C2 






X 





N5 8-34 126 








X 



N68- 34246 


' X 

'fr : 

X 



X 




W68-36931 


■ 0 

0 


0 


X 




N6 9 - 1 0 1 2 5 



L 




X 




A-5 


























RECON PROBLEM ITOMBjS* 1: RECALL A3E R5LLV ALCx. REPOST 


HIT ACCESSION 
KJUHBERS 

PJC 

AWL 

CL.N 

MEN 



FuS 

Q 


SEN 

1 

H6 9-1 1537 


0 

X 

X 

X 






069-16C87 
769-1 9061 
769-19416 



X 



X 

X 




769-20041 

769-22873 

769-23202 

0 

X 

X 

X 

X 

X 

X 




769-23741 
76 9 — 23 8 95 
76 9- 25932 


X 

X 




X 

X 


X 

769-27484 
769-27 6 34 
769-28326 

X 




X 

X 

X 

X 



W6O28506 

X 

X 

X 


X 


X 

0 


X 

769-31 076 


'X 



X 






769-31853 

769-31856 



X 


X 



0 


X 

769-31865 
769-31 868 


0 

X 

0 

X 

0 


X 

0 



769-34261 







X 




769-37466 

769-37972 





X 


X 



X 

N69-3R955 



X 


X 





X 

769-396C4 
769-4 1 046 

X 

«— • • 




0 







m 

0 

X 







769-41211 

770-1 3584 



X 




X 




770-13942 


X 



X 


X 




770- 14448 
*-0-17108 


X 



X 


X 




770-1 7426 
770-17760 

0 




X 

X 





770-2 01 S3 
370-24782 


X 



X 


X 




370-24798 

370-26221 



0 



X 

X 




770-27958 

X 

X 

X 


X 


X 

0 



3 7 0-33736 

0 




X 


X 




N7Q-35009 





X 






H7C-35160 
37039550 
770—41 149 
770-42743 


X 

0 


X 


X 



X 

771—1 0210 “ 

771-10354 



X 


X 



X 


X 

771-11 155 
77 1 - 1 1 1 6 1 

X 

X 





X 

X 


X 

771-11164 








0 


1 

771-1 1 166 

X 










: 7 1-11167 


X 








1 









































HECOR PBCBLEK NUSBEH 1: BECALL ARE &.ELEVANCE SEP OAT 


HIT ACCESSION 
-OHBERS 

PJC 

' 

All 

CLN 

Q 

T BP 


£03 

WHS 

w’W S 2 

■ 

17 1-11171 


0 

0 


X 






171-13090 




X 







17 1=13532 







X 




171- 13732 


X 



X 


c 

•o 


X 

171-16135 










X 

171=16146 










X 

171-16168 




0 






X 

17 1=18402 


X 

X 


X 








■ X 

~T3 


X 






17 1-21304 



0 

0 

X 






171=21590 



X 


X 


0 

0 



171-22607 








X 



171=24809 










X 

171-25960 





X 






171=26475 








X 



171-26774 






0 


X 



171=26904 





X 



X 



171-27545 








X 



171-27551 


X 

X 


X 


X 

0 



171-27553 




I 


X 





W7 1-27556 






X 





171-28731 




X 







171=29217 


V 

G 

X 







W7 1-30 176 



X 








171-31570 







X 




171 = 31 790 


X 





X 

0 



171-32422 



X 








171=32636 




X 







171-32997 







X 




171=33094 






X 





171-34110 







X 




171—34 5 67 



! o 




X 

0 



171=35494 

0 




X 

■ x 

X 

0 



171-36998 





X 






172-11523 





X 






17 2-12301 










X 

17 2- 12206 










X 

172-12312 



0 

X 

G 






172-13364 

X 



1 

1 X 



X 



172=15912 




1 



X 




172-16105 

X 




X 



X 



172-17270 


X 









172-17298 







X 




172-173C6 








X 


0 

172-17320 


X 



' 





X 

172-18346 



X 








172-18360 





X 

X 





172-18438 






X 





172-18439 






X 





172-18442 

1 1 






X 




































RECON PROBLEH MOUSES Is RECALL AND RELEVANCE REPOST 


— — - 

HIT ACCESSION 
VTCRBERS 

PJC 

Q 

CLN 



1 

5FS 

1 

— 

EOS 

j 

ft HS2 

I 

SBO 

1 

S72-18632 






— 1 

i 

X 




17 2—19449 





X 

i 




f 

•;72-19516 






X 





; i' 2~ 1 9 5 1 8 








X 



■: • 2-20 0 32 



X 








H72-21357 



X 


X 






H7 2—22 309 







X 




072-22365 



X 








172“ 22435 






A 





K72-24403 




X 







172-26290 




X 






X 

N72-27406 




X 







5272 — 274 38 



X 








H72-284C3 




X 







’75 -29324 

0 

0 


X 

0 


■0 




•7 2-29 379 



X 







1 

i?V2-2 94C6 







X 




"72- 29473 






x l 





r:72= 30247 







X 




i'7 2-30 3 82 

X 




X 


X 




R7 2- 3 2 334 


X 








c 

"72-33784 

X 



1 

X 


X 




pi"' 3- 10 484 



X 


X 






73-12425 




I 




X 


I 

■ 3- 3 3 32 

X 




X 





A 

i“ ‘ii 6 1 60 



X 








■73-163C3 







X 




£73-16358 




0 



X 



0 

773=* 16 387 


X 

0 

X 



X 




"73-16358 




X 
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